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Abstract 



We report measurements for two-body charmless hadronic B decays with an r]' 
meson in the final state. Using 11.1 x 10^ BB pairs collected with the Belle detector, 
we find BF{B+ r]'K+) = (79;j?±9) x 10"^ and BF{B^ r]'K^) = (55l}^±8) x 
10~^, where the first and second errors arc statistical and systematic, respectively. 
No signal is observed in the mode B^ V^^) ™d we set a 90% confidence level 
upper limit of BF{B~^ — > r/'7r+) < 7 x 10~^. The CP asymmetry in B^ rj' 
decays is investigated and a limit at 90% confidence level of —0.20 < Acp < 0.32 
is obtained. 



PACS: 13.25.Hw, 14.40.Nd 
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Charmless hadronic B decays provide a rich ground for studying the mech- 
anisms of B meson decay and the phenomenon of CP violation. The decay 
B —>■ Tj'K is an example of such a charmless decay with an unexpectedly 
large branching fraction [1]. Within the framework of the Standard Model, 
the B — > Tj'K decay proceeds primarily through 6 — > s penguin diagrams with 
a contribution from the b ^ u tree diagram. Recent theory calculations [2,3] 
underestimate the measured decay rate [4] published by the CLEO collabora- 
tion. If the large branching fraction persists after more precise measurements, 
we will likely need an additional SU(3)-singlet contribution [5] or new physics 
beyond the Standard Model to explain it. Moreover, if the unitarity trian- 
gle angle 03 (or 7), defined by Sirg{^^^^^), is greater than 90 degrees, as 

suggested from interpretations of S — > K7r,7i7i results [6,7] under the factor- 
ization assumption, B~^ r]'K~^ will be enhanced relative to rj'K^ [2,8]. 
Although expected to be small [9] , it is also of interest to examine the direct 
CP asymmetry in B^ — > r]'K^ decays since new physics may contribute. 

In this paper we report on measurements of the branching fractions of B 
mesons decaying to f]'K~^, r/V"*", and ri'K^ final states, where only the Kg 
TT+TT" transition is considered for K^. Inclusion of charge conjugate modes is 
implied unless exphcitly stated otherwise. The results are obtained from data 
collected by the Belle detector [10] at the KEKB asymmetric e'^e~ storage 
ring [11]. The data sample corresponds to an integrated luminosity of 10.4 fb^^ 
and consists of 11.1 million BB pairs at the T(4S') resonance. The branching 
fractions are calculated assuming that B^B~ and B'^B'^ are produced equally. 

A detailed description of the Belle detector can be found in Ref. [10]; here 

we only describe briefiy the parts used in this analysis. Charged tracks are 
reconstructed inside a 1.5 T solenoidal magnet with a three layer double- 
sided silicon vertex detector (SVD) and a central drift chamber (CDC) that 
consists of 50 layers segmented into 6 axial and 5 stereo superlayers. The CDC 
covers the polar angle range between 17° and 150° in the laboratory frame and, 
together with the SVD, gives a transverse momentum resolution of {(Tpt/ptY — 
(0.0019 + (0.0030)^, where pt and are in GeV/c. Charged kaon and pion 
identification is performed using a combination of three devices: an array of 
1188 aerogel Cerenkov counters (ACC) covering the momentum range 1-4 
GeV/c, a time-of-flight scintillation counter system (TOP) for track momenta 
below 1.5 GeV/c, and dE/dx information from the CDC for particles with very 
low or high momenta. Situated between these devices and the solenoid coil 
is an electromagnetic calorimeter (ECL) consisting of 8736 CsI(T£) crystals 
with typical cross-section of 5.5 x 5.5 cm^ at the front surface and a depth of 
16.2X0. The ECL provides a photon energy resolution of ((Je/£')^ = 0.013^ 
(0.0007/£;)2 + (0.008/£;V4)2^ ^ 

Charged tracks are required to come from the collision point and have trans- 
verse momenta pt above 100 MeV/c. These charged tracks are then refitted 
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with their vertex position constrained to the run-averaged profile of B meson 
decay vertices in the transverse plane. For rj' p°7 decays, in order to reduce 
backgrounds, the minimum pt requirement is increased to 200 MeV/c. 

Charged K and tt mesons coming directly from two-body B decays are iden- 
tified by combining K/tt probabilities from the CDC (dE/dx) and the ACC 
to form a K^n) likelihood Lk{Lj^). As these mesons have momenta above 1.5 
GeV/c in the laboratory frame, TOF information does not provide any dis- 
crimination. Discrimination between kaons and pions is then achieved through 
the hkelihood ratio, Lk/{Lt^ + Lk)- The performance of hadron identification 
is studied using a high momentum D*'^ data sample, where D*'^ — > D^tt'^, 
K~7i'^. Selected K and vr tracks are required to be in the same kine- 
matic region as those from two-body B decays. We measure the pion and 
kaon identification efficiencies to be (92.4 ± 2.4)% and (84.9 ± 2.1)%, respec- 
tively. The rate for true pions to be misidentified as kaons is (4.3 ± 0.4)% 
while the rate for true kaons to be misidentified as pions is (10.4 ± 0.6)%. 
For charged pions from rj' decays, tracks identified to be highly kaon-likc (in- 
cluding TOF information) are rejected. This loose kaon rejection requirement 
is studied using Kg — > tt'^tt' events. The typical efficiency for charged pions 
is (98.2 ± 1.0)%. Kg candidates are reconstructed by constraining a pair of 
oppositely charged tracks with a common vertex. This vertex is required to be 
distinct from the collision point and consistent with the Kg flight direction. 
The invariant mass is required to be within ±30 MeV/c^ of the nominal Kg 
mass. 

Two channels have been used for t]' reconstruction: rj' — > rjn'^Ti'.ri 77 and 
7]' — ^ p°7. Candidate photons from 77 [t]') decays are required to be isolated 
and have energies exceeding 50 (100) MeV. To reject soft photon backgrounds, 
— > 77 candidates are selected with | cos6'*| < 0.97, where 9* is the angle 
between the photon direction in the 77 rest frame and the 77 momentum. The 
momenta of 1] candidates with invariant mass between 500 and 570 MeV/c^ 
are recalculated by applying an 77 mass constraint. We require 7r"'"7r~ pairs from 
the rj' — > ^"7 decay to be two oppositely charged tracks having an invariant 
mass between 550 and 920 MeV/c^. Fig. 1 shows the mass distributions for 77' 
candidates. The reconstructed mass resolutions are 12 MeV/c^ for 77 —> 77, 2.7 
MeV/c^ for r]' ri%^T:~ , and 8.8 MeV/c^ for 7/' p^'-y. Candidate r]' mesons 
are required to have a reconstructed mass within 3(7 of the nominal r]' mass. 

Candidate B mesons are identified using the beam constrained mass M^c = 



V -E'beam " ^nd the energy difference l^E = £;B--Ebeam, where -Bbeam = 5.29 
GeV, and Pb and Eb are the momentum and energy of a S candidate in the 
T(4S') rest frame. In the B^ rj'h"^ {h = K,7r) study, Eb is computed with 
a kaon mass hypothesis for h~^, which results in a shift of +44 MeV in AE for 
tj'tt^ events. This shift in AE provides additional discrimination between the 
rj'K^ and 77'7r"'" final states. The parameterizations of the signal in M^c and AE 
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are determined by a GEANT [12] based Monte Carlo (MC) simulation and 
verified using a sample of D^n^, K^-k^-k^ events. The Gaussian 

width of the signal in M^c is about 2.9 MeV/c^, and mainly comes from the 
beam energy spread. The A£' distribution is found to be slightly asymmetric 
with a small tail on the lower side due to 7 interactions with material in front 
of the calorimeter and shower leakage out of the back side of the crystals. 
This AE distribution is modeled by a sum of two Gaussians, one of which 
is asymmetric. The signal region is defined as Mf,c > 5.27 GeV/c^ and —0.10 
GeV </^E < 0.08 GeV. Events located in the region M^c < 5.265 GeV/c^ are 
defined as sideband events and are used for background studies. Events with 
Mfec > 5.2 GeV/c^ and \/S.E\ < 250 MeV are selected for the final analysis. 

The dominant background for two-body B decay events comes from the e^e~ — 
qq continuum. In order to reduce this background, several shape variables are 
chosen to distinguish spherical BB events from jet-like continuum events. The 
variable 6^ is defined as the angle between the candidate 1]' direction and the 
thrust axis formed by the momenta of particles not from the B candidate. The 
continuum background, which accumulates mainly very near | cos6't| = 1-0, is 
first reduced by requiring | cos^t| < 0.9. Two other variables used are 9b, the 
angle between the B fiight direction and the beam axis, and S± [13], the scalar 
sum of the transverse momenta of all particles outside a 45° cone around the 
candidate rj' direction divided by the scalar sum of their momenta. Further- 
more, we introduce a set of variables inspired from the Fox- Wolfram moments 
[14], defined as: 

j^so ^ E»,fc \Pi\\Pk\Pi{cos9ik) ^ Eij \Pi\\Pj\Pi{coseij) 

' Ei,k\Pi\\Pk\ ' ' Ei,j\Pi\\Pj\ 

where p indicates particle momentum. Pi is the Legendre polynomial of Ith 
order, k is either 77' or h from the B candidate, and i, j enumerate all remaining 
photons and charged particles in the event. Since Rf, Rf, and are found 
to be correlated with Mbc, we do not use them. We combine the other five 
variables (/ < 4) together with cos^r and S± to form a Fisher discriminant 
F, 

F= Y.<^lRT+ E Ai?r + Cl|cOSeT|+C2-S^, 

l=2,A i=2,3,4 

where a;, /?;, Ci and C2 are determined by optimizing the separation between 
BB events and continuum events. For the channel with p7 in the final state, 
additional discrimination is gained by using the helicity variable if, which is 
the cosine of the angle between the vr"*" momentum direction in the p rest frame 
and the p momentum direction in the rj' rest frame. Although there is a small 
non-resonant contribution in the ?]' — > 7r"'"7r^7 process, experimental data [15] 
indicate that the helicity distribution can still be described by 1 — H"^. 
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The variables cosOb, F and H (for the pj channel) are found to be inde- 
pendent. The probability density functions (PDF) for these variables are ob- 
tained using MC simulations for signal, and sideband events for qq background 
(see Fig. 2). These variables are then combined to form a likehhood ratio 
LR — Lg/ {Lg + Lqq), where Ls(qq) is the product of signal (qq) probability den- 
sities. Since each channel has a different background, we optimize the LR re- 
quirement mode by mode by studying the signal significance {Ns / V Ns + Nb) 
using both MC and data samples, where Ns and Nb are signal yields and back- 
ground yields, respectively. For instance, a loose LR requirement {LR > 0.4) 
for the rjn'^n' mode keeps 83% of the rj'K'^ signal and reduces 74% of the 
background, while for the pj mode a tighter requirement {LR > 0.6) is used 
with a signal efficiency of 70% and a background rejection of 88%. The ef- 
fect of these requirements is studied by comparing — > D^7r~^ in data and 
MC with different values of LR. The background from b ^ c transitions is 
neghgible as determined by MC study. 

Fig. 3 shows the M^c and AE distributions for the combined rjTnT and p7 sam- 
ples (and rinn separately) in the r]'K~^ ,r]'7r~^ and rj'Kg decay modes. Events in 
the Mbc {AE) plots are required to be in the AE (M^c) signal region after all 
selection criteria are applied. Clear signal peaks appear in both the rj'K'^ and 
rj'Kg channels. The Mbc and AE distributions in each sample are fitted simul- 
taneously with signal and background functions using an extended unbinned 
maximum likelihood fit. For N input candidates, the likelihood is defined as 

-{Ns+Nb) N 

L{Ns, Nb) = l[[NsPs,{Mbc, ^E) + NbPbAMi>c. ^E)], 

-'^ • 1=1 



where P5. and Pb^ are the probability densities for event i to be signal and 
background for variables M^c and AE, respectively. In the extended maximum 
likelihood (ML) fit, the extracted yields for signal {Ns) and for background 
{Nb) are considered separately to follow the Poisson statistics. In this defini- 
tion, the sum of Ns and Nb equals the number of input candidates N when 
the likelihood is maximized. The M^c background shape is modeled with a 
smooth function [16] with parameters determined using events outside of the 
AE signal region. The AE background shape is modeled by a linear function 
with the slope determined from the sideband data and cross-checked using MC 
events. Since significant signal peaks are observed in the •q'K'^ channel, we can 
expect contamination in the Ty'Tr"*" signal region from rj'K'^. Therefore, an rj'K'^ 
signal shape is added in the AE PDF for ^^V"*". The final signal yields are then 
obtained by this two dimensional (2-D) fit with the statistical significance (S) 

defined as y^— 2 ln(i/o/ -^max), where Lq and Lmax denote the likelihood values 
at zero signal events and the best fit numbers, respectively. The second and 
third columns of Table 1 fist the fit yields and their significances. 
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The systematic error for the signal yield is estimated by varying each param- 
eter of the fit functions by ±1(7 from the measured values. The shifts in signal 
yield are then added in quadrature. In order to study intrusions into the sig- 
nal regions from other rare B decays, a large MC sample of all known rare B 
decay processes has been generated. No significant contributions are found. 

Signal efficiencies are first obtained using MC simulations and corrected by 
comparing data and MC predictions for other processes. The tracking effi- 
ciency is studied using high momentum D, 77 and K*{892) samples. The 77 
reconstruction efficiency is verified by measuring the branching ratio of two 
decay channels, K'tt+tt^ to K-tt+, and four i^*(892) decay 

channels, ir' , , Kg7r~^ , and Kgii^. The simulation of low momentum 

photons is further tested by comparing the decay angular distribution of r] for 
data with MC predictions. The systematic errors on the charged track recon- 
struction efficiencies of rj' — > rjir'^Tr' and rj' — > p7 are estimated by comparing 
the ratio oi rj ^ 7r"'"7r~7r° to 77 77 in data with MC expectations. The re- 
construction of high momentum Kg is studied using the ratio of Kg7r~^ 
to — > K~7r~^7r~^. The final systematic errors, including contributions from 
reconstruction efficiency, hadron identification and 2-D fits, are estimated to 
be 12% for the rj'K'^ and rj'n'^ modes, and 15% for the rj'Kg mode. 

Table 1 summarizes the fit results for each reconstructed decay channel. The 
systematic errors for the branching fractions combine the above systematic 
errors with that from the number of BB events. Results from the r]7r~^7r~ 
and pj channels are then combined by adding the —2\aL(BF) functions of 
branching fractions with appoximating N = Ns+Nb and extracting the values 
and la deviations at maximum L{BF). Since no significant signal is seen in 
the rj'-K^ mode, a 90% confidence level (C.L.) upper limit is given by finding 
the branching fraction that corresponds to 90% of the integral of L[BF). The 
final upper limit is then computed by adding one standard deviation of the 
systematic error. The final branching fractions, significance, upper limit, and 
theoretical predictions are listed in Tabic 2. With 11.1 million BB events, we 
measure the branching fractions to be (79l|^ ± 9) x 10^^ for B^ rj'K^ 
and (55l}^ ± 8) x 10"^ for 5° ^ r]'K^. The first errors are statistical and 
the second systematic. The 90% C.L. upper hmit for B^ — > t^'tt^ is 7 x 10~^. 
Our results are consistent with previous measurements given by CLEO [4], 
but larger than theoretical predictions [2]. The results are also compatible 
with the preliminary results from BABAR [17]. The upper limit on t^'tt"*" is 
currently the most restrictive experimental result. 

To study charge asymmetry, the 'q'K^ sample is divided into two subsamples: 
r]'K^ and r]'K~. The 2-D fit in the M^c vs. AE plane is performed for each 
subsample. The fitted number of signal events in the rj' rinn, and rj' p'-f 
modes are 18.2^|i, and 13.9l|2 for B+ decays, and 10. 71^^, and 27.9^^;^ for 
B~ decays, respectively. The number of produced S+ and B~ are obtained 
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by maximizing the product of the hkehhoods for each submode. The number 
of produced events is 398t^[! for B+ r]'K+ and 450^^^ for r]'K-; the 

errors are statistical only. The CP asymmetry {Aqp) is calculated as {N{B~) — 
N{B~^))/(N(B~) + N(B~^)). Since the systematic errors on rj' reconstruction 
and the number of BB events cancel in the ratio, the systematic uncertainty of 
Acp comes mainly from the reconstruction efficiency of charged kaons and the 
2-D fit. The asymmetry in the K"^ efficiency is studied using inclusive charged 
kaons. The latter is measured by varying the parameters of the fit functions. 
We find the systematic errors for Acp are 0.01 for reconstruction and 0.01 
for the 2-D fit [18]. The Aqp for the B^ — > rj'K'^ decay is finally measured to 
be +0.06 ± 0.15 ± 0.01, which corresponds to -0.20 < Acp < 0.32 at the 90% 
confidence level. 

In summary, we have searched for charmless hadronic B decays with rj' mesons 
in the final state. Our results confirm that the branching fractions of B~^ — > 
ri'K~^ and B^ rj'K^ are large. The branching fraction of B^ t^V"*" is less 
than 7x 10"^ at the 90% C.L. With about 70 B^ r]'K^ events, no significant 
charge asymmetry is observed. Our value for B — > r]'K~^ is somewhat larger 
than that for B rj'K^, as predicted under the factorization assumption, 
but the difference is not statistically significant. If this difference is confirmed 
with better precision, it will, along with a measurement of time dependent CP 
asymmetry in B^ rj'K'^, provide information to understand the underlying 
dynamics of i? — > rj'K decays. 
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Tabic 1 

Summary of results for each channel listed in the first column. The measured signal 
yield (Ns), statistical significance (S), reconstruction efficiency (e), total efficiency 
including the secondary branching fraction {Bg), and the measured branching frac- 
tions are shown. The branching fractions are calculated by assuming that B^B~ 
and B^B^ are produced equally from T(4<S') decays. Uncertainties shown in 2nd and 
6th columns are statistical only. 



Mode 



Ns 



S e(%) eS,(%) EF(10-6) 



n' 


28.9111 


9.4 


21.7 


3.78 


69lit 


n' K+ 


42.511^ 


7.5 


14.2 


4.18 


921?^ 






0.0 


23.7 


4.11 








0.0 


15.4 


4.55 






6.4ir, 


3.5 


20.8 


1.25 


4611 






4.0 


11.5 


1.16 


7911 



Table 2 

Combined branching fractions (BF) or 90% C.L. limit, significance (E) of Belle, 
CLEO [4], BABAR [17] and theoretical expectations [2,3]. The branching fractions 
are in units of 10~^. 



Mode 



This measurement 



CLEO BABAR Theory 



B+ 
B+ 



r]'K+ 



79tll ± 9 



< 7 



55 



+19 
-16 



12.0 80l^° ± 7 62 ± 18 ± 8 



0.0 



< 12 



5.4 89tll ± 9 



< 112 



21-53 

1-3 
20-50 




0,92 0.94 0.96 0,98 

rj'{rj7\7\) moss (GeV/c^) 



0,92 0.94 0,96 0,98 

rj'{py) moss (GeV/c^) 



Fig. 1. Reconstructed mass spectra for rj' — ■qiv'^'K (left) and rj' — p7 (right) for 
events in the Mf^-l^E signal region after applying all analysis requirements. 
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-1 -0.5 0.5 1 0.2 0.4 0.6 0.8 
H LR 



Fig. 2. Event shape distributions for (a) cosOb, (b) F, (c) H and (d) Li? for the 
fry mode. Histograms represent the signal distributions (MC), while solid points are 
from data outside the signal region. Superimposed in (a), (b) and (c) are the PDF 
par ameterizations . 
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5.2 5.25 5.3 -0.2 0.2 



Mf^^iGeY/c^) A£(GeV) 

Fig. 3. Mi)c and AE projections for r]'K^, rj-K^ and tI events. The shaded his- 
tograms correspond to the r/' — > r/7r"'"7r~ mode while the histograms are for all the 
modes combined. The superimposed curves show the fits to M{,c and IS.E. Solid 
curves are for signals plus backgrounds and dashed curves are for backgrounds only. 
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Fig. 4. Mfec and AE projections for r]'K^ and rj'K^ events. The shaded histograms 
correspond to the ij' — > r]i:^i:~ mode while the histograms are for all the modes 
combined. The superimposed curves show the fits to M^c and AE. Solid curves are 
for signals plus backgrounds while the dashed curves are for backgrounds only. 
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